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In  an  effort  to  model  cortical  networks  and  emulate  higher-level  brain  function, 
especially  in  the  recognition,  classification,  and  learning  of  spatio-temporal  signals  of  the 
kind  occurring  in  natural  and  artificial  settings,  we  have  been  investigating  the  dynamics  of 
a  new  class  of  networks  composed  of  parametrically  coupled  bifurcation  processing 
elements.  The  spatio-temporal  signals  of  particular  interest  to  us  are  of  the  variety 
produced  by  sensor  arrays  in  radar,  sonar  and  ATR  as  result  of  relative  motion  between  the 
sensor  platform  and  the  scattering  object.  The  bifurcation  processing  elements  in  these 
networks  are  logistic  maps  representing  netlets  or  neuronal  assemblies  of  the  cortex. 

In  the  preceding  quarterly  report  evidence  was  presented  in  support  of  the  hypothesis 
that  the  basic  functional  unit  in  the  cortex,  the  seat  of  higher-level  brain  function,  maybe 
mathematically  modeled  by  a  bifurcation  processing  element:  a  parametrically  driven 
noninvertible  map  on  the  unit  interval  such  as  the  logistic  map  or  the  sine-circle  map.  Such 
simplifying  abstraction  seems  to  capture  functional  attributes  of  cortical  units  as  seen  in 
numerical  simulations  of  a  network  of  parametrically  coupled  bifurcation  processing 
elements.  Specifically,  a  bifurcation  processing  unit  is  functionally  complex  in  that  it  can 
assume  any  of  a  number  of  qualitatively  different  functional  modalities  that  include  regular 
(fixed-point,  periodic  (period-m))  and  irregular  (intermittant,  chaotic)  activity  and  can 
bifurcate  (rapidly  switch)  among  them  depending  on  the  net  input  to  the  unit  from  other 
units.  The  network  studied  is  shown  to  exhibit  behavior  remarkably  similar  to  that  seen  in 
functional  magnetic  resonance  imaging  (fMRI)  of  brain  function  suggesting  that  it  offers  an 
efficient  tool  for  modeling  and  studying  cortical  dynamics  and  higher-level  brain  function. 

Under  extrinsic  dynamic  input  the  network  studied  shows  input-specific  isolated 
clustering  of  activity  analogous  to  the  “hot-spots”  of  activity  revealed  by  fMRI  of  brain 
activity  of  individuals  subjected  to  sensory  stimulus  or  engaged  in  mental  activity 
associated  with  solving  an  assigned  cognitive  task. 

Simulation  results  also  predict  that  within  the  isolated  active  regions  (hot-spots) 
neuronal  groups  assume  complex  temporal  structures  that  describe  fixed-point,  periodic, 
quasi-periodic,  intermittant,  or  chaotic  orbits  that  serve  apparently  to  further  characterize  the 
input  stimulus.  Unfortunately  the  temporal  resolution  of  fMRI  is  not  sufficient  to  reveal 
such  temporal  effects  in  actual  observations.  However  other  faster  emergent  functional 
brain  imaging  techniques  such  as  magnetoencephalography  (MEG)  employing  arrays  of 
superconducting  quantum  interference  devices  (SQUIDS)  or  optical  recoring  techniques 
may  be  of  help  in  testing  this  prediction.  Positive  verification  of  this  prediction  could  have 
far  reaching  implications  for  brain  modeling  and  help  elucidate  the  role  of  such  possible 
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temporal  encoding  in  brain  function  furnishing  thereby  guidelines  for  the  design  of  artificial 
intelligent  systems. 

Interestingly  the  network  studied  is  found  to  exhibit  the  above  behavior  only  when 
several  salient  or  plausible  attributes  of  thalamo-cortical  organization  and  interaction  were 
incorporated  in  it.  This  included  predominance  of  local  connections,  nonlinear  (activity- 
dependent)  coupling  between  units,  ability  to  accept  extrinsic  spatio-temporal  input,  and  the 
gradual  turning  over  of  control  of  network  dynamics  from  extrinsic  (sensor  driven)  control 
to  intrinsic  (internal  feedback)  control. 

We  are  continuing  to  examine  the  implications,  predictive  power,  and  applications  of 
this  class  of  networks  which  because  of  their  ability  to  classify  spatio-temporal  input 
patterns  are  of  interest  in  ATR  and  other  areas  such  as  speech  processing.  However  most 
of  the  effort  during  the  period  of  this  report  was  directed  to  the  issue  of  hardware 
implementation  of  networks  of  parametrically  coupled  logistic  processing  units  that  can 
furnish  the  speed  needed  in  practical  applications.  To  this  end  we  concentrated  on 
determining  the  plausibility  of  building  logistic  processing  elements  that  can  be  used  as 
building  block  in  hardware  implementations  of  parametrically  coupled  networks  of  logistic 
(bifurcation)  processing  units. 

In  previous  work  we  have  demonstrated  that  a  programmable  unijunction  transistor 
(PUT)  circuit  with  cosinusoidal  modulation  of  the  PUT’s  extinction  voltage  acts  as  a  sine- 
circle  map  which  exhibits  bifurcation  between  distinct  modalities  and  route  to  chaos. 

During  the  period  of  this  report  we  attempted  to  answer  the  following  question: 

Can  we  obtain  a  spiking  oscillator  based  on  the  PUT  for  which  the 
logistic  map  emerges  as  the  relationship  between  consecutive  spikes? 

If  this  is  possible,  which  modifications  must  be  made  in  the  original 
circuit  (i.e.,  the  circuit  that  implements  the  sine-circle  map  bifurcating 
processing  element)  in  order  to  obtain  this  new  dynamical  behavior? 

To  date  the  work  performed  shows  that  by  using  the  same  kind  of  circuit  employed 
for  the  earlier  implementation  of  the  sine-circle  map  bifurcating  processing  element  as  well 
as  by  appropriately  changing  the  waveform  of  the  periodic  signal  that  is  used  to  drive  the 
PUT,  the  logistic  recursion  can,  in  fact,  be  obtained.  This  finding  is  being  examined 
further  because  it  seems  to  suggest  that  the  PUT  circuit  with  periodic  modulation  of 
extinction  voltage  may  be  useful  for  producing  arbitrary  maps  on  the  interval  and  thus 
bifurcation  processing  elements  with  arbitrary  desired  properties  by  merely  altering  the 
shape  of  the  periodic  modulation  waveform.  If  confirmed  this  finding  could  be  important 
for  dynamical  computing  employing  timing  or  phase  as  variable  and  in  encryption,  secure 
communications  and  secure  transponders. 
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Also  during  this  period  N.  Farhat  (PI)  participated  in  the  following  meetings: 


1997  Annual  Meeting  of  the  Optical  Society  of  America  fOSA’97) 
where  he  presented  two  papers: 

1 .  “Neuroholography:  a  Possible  Link  Between  Holography  and  Cortical  Information 
Processing.” 

2 .  “Biomorphic  Networks  for  Invariant  Feature  Extraction.” 

He  also  participated  in  the: 

1 997  International  Symposium  on  Nonlinear  Theory  and  Applications  (NOLTA’97) 
by  presenting  a  paper  entitled: 

“Dynamical  Networks  with  Bifrucation  Processing  Elements.”  The  paper  was 
published  in  the  Symposium  proceeding,  (copy  attached). 

Also  during  this  period  past  work  done  under  this  grant  was  referred  to  in  a  special  news 
report  in  Science  under  the  title  “A  Subtler  Silicon  Cell  for  Neural  Networks.”  (copy 
attached). 


1997  International  Symposium  on  Nonlinear 
Theory  and  its  Applications  (NOLTA’97) 
Honolulu,  U.S.A.,  Nov.29-Dec.  2, 1997 
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PROCESSING  ELEMENT 

Nabil  H.  Farhat 
University  of  Pennsylvania 
Electrical  Engineering  Department 
200  South  33rd  Street 
Philadelphia,  PA  19104,  USA 
e.m.  farhat@pender.ee.upenn.edu 


Abstract:  We  introduce  the  concept  of 
parametrically  and  nonlinearly  coupled  network 
of  bifurcation  processing  elements  that  can  be 
driven  by  static  or  dynamic  input  patterns.  The 
network  is  biologically  inspired,  computes  with 
all  three-types  of  attractors  and  offers  a  unique 
tool  for  the  modeling  and  study  of  cortical 
networks  and  higher-level  brain  function. 

1.  Introduction:  There  is  considerable 
evidence  that  the  basic  functional  unit  for  higher- 
level  processing  in  the  cortex  is  the  netlet  or 
neuronal  assembly  (pool  or  group)  [1]-[10].  This 
evidence  includes  extensive  analytical  and 
modeling  work  of  netlets  carried  out  in¬ 
dependently  by  several  groups  in  the  past. 
Nearly  all  that  body  of  work  points  to  the 
possibility  that  netlet  dynamics,  may  be 
adequately  described  by  the  discrete  time 
evolution  of  the  activity  A(n),  which  is  the 
percentage  of  neurons  in  the  netlet  active  at  time 
n.  Plots  of  A(n+1)  vs.  A(n)  obtained  under  a 
range  of  circumstances  and  assumptions  are  found 
to  invariably  resemble  a  distorted  version  of  the 
quadratic  or  logistic  map,  a  nonlinear  iterative 
map  on  the  unit  interval  that  exhibits  complex 
orbits  depending  on  the  value  of  a  nonlinearity 
(control  or  bifurcation)  parameter  [11],  The 
similarity  between  the  netlet’s  return  map 
A(n+1)  vs.  A(n)  and  that  of  the  logistic  map  has 
also  been  noted  by  Harth  [10]  who  also  mentions 
that  complex  and  unpredictable  sequences  A(n) 
were  observed  in  some  of  their  early  simulations 
of  netlets  suggesting  that  certain  regions  of  the 
netlet's  parameter  space  can  lead  to  observation 
of  chaos  in  addition  to  the  periodic  and  fixed 
point  modalities  they  usually  observed. 

In  light  of  this  evidence  we  have  conjectured 
that  cortical  networks  can  be  modeled  aid 
numerically  studied  in  an  efficient  way  by  means 
of  parametric^  ly  coupled  populations  of  logistic 
processing  elements  [12],  To  test  this  conjecture 
we  have  studied  the  dynamics  of  such  a  network 


when  it  is  subjected  to  dynamic  input:  external 
stimulus  patterns  that  changed  in  time.  The 
networks  we  study  differ  from  coupled  map 
lattices  (CMLs),  [13]-(14],  in  several  ways:  (a) 
The  networks  described  here  employ  parametric 
rather  than  the  diffuse  coupling  used  in  CMLs, 
(b)  The  coupling  is  nonlinear  representing  the 
possibility  that  the  interaction  between  cortical 
netlets  can  depend  on  the  activity  of  the  netlets 
and  on  the  number  of  active  fibers  connecting 
one  netlet  to  another,  (c)  Parametrically  coupled 
logistic  nets  (PCLNs)  can  be  externally  driven  by 
dynamic  or  static  patterns,  or  by  composite 
patterns  that  are  partially  time  varying  aid 
partially  stationary,  (d)  In  the  PCLN,  control 
over  network  dynamics  is  gradually  handed  over 
from  initially  entirely  extrinsic  control  to 
eventually  entirely  intrinsic  control.  This 
gradual  transfer  of  control  over  network  dynamics 
from  extrinsic  to  intrinsic  is  biologically 
plausible  and  is  inspired  by  the  remarkable 
biophysical  observation  made  by  Freeman  and 
coworkers  [15]  regarding  gradual  disappearance  of 
the  trace  of  a  sensory  stimulus  applied  to  the 
olfactory  bulb  of  rabbit  as  it  was  followed  deeper 
in  the  sensory  cortex  where  it  was  found  to 
eventually  vanish  in  a  sea  of  intrinsically 
dominated  activity.  Similar  behavior  has 
apparently  been  observed  by  Freeman's  group  in 
other  sensory  modalities. 

The  preceding  remarks  suggest  that  networks 
of  parametrically  coupled  logistic  maps  offer  an 
effective  way  to  study  the  functional  complexity 
of  cortical  networks  in  order  to  understand  the 
way  they  perform  higher-level  functions.  Such 
higher-level  functions  are  beyond  the  capabilities 
of  present  day  sigmoidal  networks,  and 
incorporating  them  in  artificial  network  offers  a 
way  for  increasing  their  processing  power  and  for 
widening  their  scope  of  application. 

2,  The  Network:  The  network  studied  is 
shown  in  Fig,  1.  It  consists  of  a  one- 
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dimensional  array  of  N  parametrically  coupled 
logistic  maps.  Parametric  coupling  means  the 
nonlinearity,  (control  or  bifurcation  parameter) 
fX\  of  the  i-th  map  is  not  fixed  but  is  modulated 
in  time.  In  the  network,  }X\  is  modulated  by 
both  extrinsic  and  intrinsic  influences  according 
to 

Mi(n)  ■  e(n)gf(n) 


1  - 


j=i-Ni 

e(n)  =6.  expC-an) 


(1) 


In  eq.  (1)  i  =  1.2....N,  the  first  term  represents 
the  extrinsic  (sensory)  input  to  the  i-th  logistic 
processing  element  or  cell,  the  second  term 
represents  the  net  intrinsic  input  to  the  i-th  cell 
through  feedback  from  all  other  cells  connected  to 
it,  n  is  discrete  integer  time,  2N[  is  the  number 
of  logistic  cells  connected  to  the  i-th  cell  i.e.  the 
number  of  cells  falling  within  a  *  connection 
radius  Rc”  that  is  taken  to  be  identical  for  all 

cells,  gf(n)  =  4(ui(n))w»  is  the  extrinsic 
(sensory)  input  to  the  i-th  cell  with  state  variable 
Uj(n)e  [0,1]  being  produced  in  the  simulation 
conveniently  by  a  sensory  logistic  map  according 
to:  uj(n+l)  =  fJ.f  Ui(n)  (l  -  Ui(n))  with 

U[(0)  =  0.5  and  fJ.?  being  a  fixed  control 

parameter  of  the  i-th  stimulus  generating  logistic 

map.  Selecting  [if  in  [0,4]  enables  the 

production  of  a  wide  range  of  stationary,  periodic 
(period-m)  or  chaotic  patterns  u[(n)  or  any 

desired  combination  of  such  patterns  on  i 

S 

depending  on  the  values  one  selects  for  /-£* . 

Thus  by  adjusting  the  control  vector  fl  of  the 
N  stimulus  generating  logistic  cells,  a  wide 
variety  of  spatio-temporal  driving  signals  can  be 
generated  and  applied  to  the  network.  The 
coupling  factor  Wj  ranges  between  0  and  For 

example  =  0  produces  g?(n)  =  4  which 

means  the  extrinsic  contribution  when  added  to 
the  intrinsic  one  tends  to  make  (n)  high  with 
the  result  that  the  i-th  processing  cell  would  tend 
to  be  chaotic.  On  the  other  hand  wj  =  00  yields 


when  t.  is  in  [O.l]  «nd  a  i  0  is  positive  real  number. 

w*  and  wf  are  nonlinear  activity  dependent  coupling  functions  with 
possible  forms  each: 


wfm 


-  quantized  version  of  (V,  levels) 


SF(jr,)  -  quantized  version  of g,  (*,)  (V*  levels) 


Fig.  1.  Parametrically  coupled  logistic  network 
(PCLN)  consisting  of  N  bifurcation 
(logistic)  processing  elements  or  cells. 
The  network  employs:  (a)  novel 
biologically  plausible  nonlinear  (activity) 
dependent  coupling  functions  between 
ceils  each  representing  a  netlet  and  (b)  a 
biologically  plausible  gradual  transfer  of 
control  over  network  dynamics  from 
initially  totally  extrinsic  (sensory) 
control  to  totally  intrinsic  control.  The 
network  can  be  driven  externally  by 
spatio-temporal  inputs  provided  by  the 
stimulus  generating  network  that 
employs  an  array  of  uncoupled  logistic 
maps  to  conveniently  produce  a  variety 
of  static,  time-periodic,  chaotic,  or 
composite  signals  made  of  any  mix  of 
these  three-types  of  signals.  The 
quantized  versions  of  the  coupling 
functions  used  to  study  the  coarse-grain 
dynamics  of  the  PCLN  (not  discussed 
here). 
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g?(n)  as  0,  because  the  state  variable  uj  of  the 

logistic  map  is  in  [0.1].  This  means  that  small 
values  of  wj  introduce  disorder  while  larger 
values  introduce  inhibition.  Similarly,  the 
quantity  gy(n)  in  eq.  (1)  represents  the  input 

from  the  j-lh  cell  to  the  i-th  cell;  it  has  a  form 
similar  to  g?(n),  namely  gij(n)  = 

4|Xj(n)]  1J  with  Cy  being  in  [0,~J  and 
Xj(n)  is  the  state  variable  of  the  j-th  logistic 
processing  cell  of  the  network  governed  by; 
Xj(n  + 1)  =  /ij(n)(l  -  Xj(n))  where  m(n)  is 
given  by  eq.  (1)  and  Xj(n)  is  also  in  [0,1]. 
Note  the  nonlinear  dependence  of  g?(n)  on  uj(n) 
and  of  g  i j  (n)  on  Xj(n)  serves  two  purposes. 

One,  it  confines  their  combined  contribution  to 

n)  to  the  allowable  range  [0,4],  and  second, 
the  values  of  v/\  and  Cy  provide  control  over 

the  level  of  excitation/disorder  on  the  one  hand  or 
inhibition  on  the  other,  that  are  injected  into  the 
dynamics  of  the  i-th  cell,  and  hence  into  the 
network  as  a  whole,  by  the  i-th  sensory  cell  or 
by  the  j-th  processing  cell  respectively.  The 
parameter  a  in  eq.  (1)  is  a  positive  real  constants 
whose  value  determines  the  speed  with  which 
control  over  the  dynamic  of  the  network  is 
handed  over  from  initially  entirely  extrinsic 
control  to  eventually  and  entirely  intrinsic 
control.  A  value  of  e  0  =  1  means  that  initially 
the  dynamics  of  the  network  are  totally  controlled 
by  the  extrinsic  (sensory)  pattern  and  a  =  0 
means  there  is  no  fading  of  the  effect  of  extrinsic 
input. 

The  behavior  of  the  PCLN  was  numerically 
studied  under  a  variety  of  conditions  and 
parameter  values  and  extremely  rich  behavior  was 
observed.  Particularly  interesting  was  the 
behavior  when  local  connectivity  was  used  where 
Nj  «  N  to  reflect  presumably  the  dominance  of 
local  inter-connection  between  netlets  of  the 
cortex.  When  self-connection  of  cells  was 
allowed  and  the  following  parameters  were  used: 
Rc randomly  selected  in  [0,1],  e0  =  1,  a  =0.1, 
w.  =  0.5,  and  random  coupling  coefficients  i.e., 
Cy  randomly  and  uniformly  selected  in  [a,b],  and 
random  initial  state  vector  X(0)  i.e.  X^O) 
randomly  and  uniformly  selected  in  [0,1],  the 


network  exhibited  isolated  clusters  of  activity  for 
values  of  the  constants  a  ranging  in  [0,0.3]  and 
b~3  which  furnished  a  mix  of  chaos  and  order 
inducing  coupling  functions.  The  form  of 
isolated  clustering,  and  the  orbits  of  cells  within 
clusters,  were  stimulus  specific  and  independent 
of  initial  state  of  the  network,  as  desired.  The 
clustering  was  relatively  rapid  occurring  usually 
within  the  Fust  100  iterations  depending  on  the 
value  of  a.  Cell  orbits  of  different  type  i.e., 
fixed-point,  period-m,  and  chaotic  can  coexist 
within  a  cluster  and  often  the  period-m  orbits  of 
cells  within  well  separated  clusters  were  not  only 
phase-locked  but  synchronized.  This  latter 
behavior  conforms  with  synchronized  oscillations 
of  local  field  potentials  observed  by  several 
workers  in  the  brain  of  cat  and  monkey  and  with 
Eckhom’s  modeling  of  that  behavior  in  networks 
of  spiking  neurons  [16]-[21].  Most  interesting  is 
the  isolated  clustering  which  is  analogous  to  the 
isolated  clusters  of  brain  activity  seen  in 
functional  magnetic  resonance  imaging  (fMRI) 
and  positron  emission  tomography  (PET)  images 
of  subjects  subjected  to  sensory  stimulus  or 
when  performing  an  assigned  cognitive  or  motor 
task. 

The  conceptual  similarity  of  the  isolated 
clustering  behavior  in  PCLNs  and  the  clustering 
of  brain  activity  seen  in  fMRI  and  PET*  raises  an 
interesting  scientific  question.  If  PCLNs  a re 
valid  models  of  cortical  nets  then  the  clusters  of 
brain  activity  seen  in  fMRI  and  PET  should  also 
exhibit  analogous  temporal  activity.  Un¬ 
fortunately  the  time  resolution  of  fMRI  and  PET 
at  present  is  too  coarse  to  discern  any  temporal 
activity  within  the  clusters  that  light-up  because 
both  measure  the  change  in  blood  flow  to  active 
brain  regions.  An  increasing  number  of  studies 
employing  PET  and  fMRI  show  however  that 
different  sensory  stimuli  or  cognitive  tasks  can 
“light-up”  the  same  brain  spot.  This  strongly 
suggests  a  role  for  temporal  encoding  to  enable 
differentiation.  It  would  be  interesting  to  see  if 
future  technological  advances  in  functional  brain 
imaging  could  provide  the  needed  temporal 
resolution  to  verify  the  prediction  of  the  PCLN. 

3.  Conclusion:  The  generally  rich  behavior 
we  observe  with  PCLNs  including  the 
remarkable  specific  behaviors  described  above 
have  no  parallel  in  sigmoidal  neural  network,  and 
apparently  also  in  coupled  map  lattices  and 
cellular  automata.  Therefore  we  believe  that  the 
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use  of  PCLNs  to  model  cortical  networks  and 
higher-level  brain  functions  provide  a  unique  tool 
for  the  development  of  intelligent  systems  that 
can  operate  in  a  natural  environment  where  time 
varying  signatures  are  the  norm  and  not  the 
exception. 
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the  simulations  referred  to  were  carried  out.  This 
research  was  supported  by  the  Office  of  Naval 
Research  under  grant  no.  N000 14-94- 1-093 1. 

5.  References: 

1.  E.M.  Harth,  et.  al.,  J.  Theor.  Biol.,  vol. 
26,  pp.  93-120,  (1970). 

2.  P.A.  Annios,  et.  al.,  J.  Theor.  Biol.,  vol. 
26,  pp.  121-148,(1970). 

3.  M.  Usher,  H.G.  Schuster  and  E.  Neibur,. 
Neural  Computation,  vol.  5,  pp.  370-386, 
July,  (1993). 

4.  G.M.  Edelman,  Neural  Darwinism:  The 
Theory  of  Neuronal  Group  Selection , 
Basic  Books,  Inc.,  Publishers,  New  York, 
(1987). 

5.  C.  van  Vreeswijk  and  H.  Sompolinski, 
Science,  vol.  274,  pp.  1724-1726,  Dec. 
(1976). 

6.  T.  Wennekers  and  F.  Pasemann, 

International  Journal  of  Bifurcation  and 
Chaos  in  Applied  Science  and 
Engineering,  vol.  6,  pp.  2055-2067, 

(1996). 

7.  T.  Wennekers,  F.  Sommer  and  G.  Palm, 

in  Supercomputing  in  Brain  Research: 
From  Tomography  to  Neural  Networks , 
H.  Hermann,  D.  Wolf  and  E.  POppel 
(Eds.),  World  Scientific,  Singapore, 

(1995). 

8.  G.L.  Shaw  and  DJ.  Silverman,  in 
Computer  Simulation  in  Brain  Science^ 
R.  Cotterill  (Ed.),  Cambridge  Univ.  Press, 
Cambridge,  (1988),  pp.  189-209. 

9.  M.  Shadlen  and  W.  Newhouse,  Opin. 
Neurobiology,  vol.  14,  pp.  569-579, 
(1*994). 

10.  E.  Harth,  Order  and  Chaos  in  IEEE  Trans, 
on  Systems  Man  and  Cybernetics,  vol. 
SMC-13,  pp.  48-55,  (1983). 

11.  R.C.  Hilbom,  Chaos  and  Nonlinear 
Dynamics ,  Oxford  Univ.  Press,  New  York 
(1994). 


12.  N.  Farhat  and  E.  del  Moral  Hernandez, 

“Recurrent  nets  with  recursive  processing 
elements:  Paradigm  for  dynamical 

computing,*’  SPIE,  vol.  2324,  SPIE, 
Bellingham,  Wash.  (1996),  p.  158-170. 

13.  K.  Kaneko,  in  Theory  and  Applications  of 
Coupled  Map  Lattices ,  K.  Kaneko  (Ed.), 
J.  Wiley,  New  York,  (1993),  pp.  1-49. 

14.  J.  Crutchfield  and  K.  Kaneko,  in 
Directions  in  Chaos ,  vol.  1,  World 
Scientific  Publishing  Co.,  Singapore, 
(1987),  pp.  272-353. 

15.  W.J.  Freeman,  Societies  of  Brains ,  LEA 
Associates  Publishers,  Hillsdale,  N.J., 
(1995). 

16.  C.M.  Gray  and  W.  Singer,  “Stimulus 
specific  neuronal  oscillations  in  the  cat 
visual  cortex:  a  cortical  functional  unit,” 
Soc.  Neurosci.,  abstr.,  404,  (1987). 

17.  _ ,  “Stimulus  dependent 

neuronal  oscillations  in  the  cat  visual 
cortex  area  17,”  Neuroscience,  vol.  22, 
[suppl.],  (1987). 

18.  C.M.  Gray,  P.  Konig,  A.K.  Engel  and  W. 
Singer,  “Oscillatory  responses  in  cat 
visual  cortex  exhibit  inter-columnar 
synchronizadon  which  reflect  global 
sdmulus  properties,”  Nature,  vol.  338, 
pp!  334-337,  (1989). 

19.  C.M.  Gray  and  W.  Singer,  “Sdmulus 
specific  neuronal  oscilladon  in  orientadon 
columns  in  cat  visual  cortex,”  Proc.  Natl. 
Acad.  Sci.,  USA,  vol.  86,  pp.  1698-1702, 
(1989). 

20.  R.  Eckhom,  M.  Arndt  and  P.  Dike, 
“Feature  linking  via  synchronizadon 
among  distributed  assemblies:  simulauons 
results  from  cat  visual  cortex,”  Neural 
Computadon,  vol.  2,  pp.  393-307, 
(1990). 

21.  R.  Eckhom  and  HJ.  Reitboeck, 
“Stimulus-specific  synchronizadon  in  the 
cat  visual  cortex  and  its  possible  role  in 
visual  pattern  recognidon  ”  in  Synergetics 
of  Cognition ,  H.  Haken  and  M.  Stadler 
(Eds.),  pp.  99-111,  Springer  Ycrlag, 
Berlin,  (1990). 


268 


S2D-3 


A  Subtler  Silicon  Cell  for  Neural  Networks 


Nature  is  the  model  for  artificial  neural  networks.  These  networks 
of  processors — either  real  or  simulated  on  a  conventional  com¬ 
puter — “learn”  from  experience  by  adjusting  the  strength  of  their 
connections,  much  like  networks  of  real  neurons.  Small  neural  nets 
have  become  commonplace,  doing  tasks  such  as  predicting  how 
stock  prices  may  fluctuate  and  recognizing  handwritten  characters. 

But  Nabil  Farhat  of  the  University  of  Pennsylvania,  Philadelphia, 
thinks  he  can  build  a  better  neural  net  by  making  its  constituent 
neurons  even  closer  to  biology. 

Neural  nets  traditionally  consist  of  so-called  sigmoidal  neurons, 

circuits  that  add  up  incoming  signals  un-  6.28  ^ — ■ -  .pw ; 

til  they  reach  a  fixed  threshold  and  then  ■  fi 

fire  themselves.  Farhat’s  so-called  bifur-  i  j 

cation  neurons,  in  contrast,  switch  be-  4.71  -  Aj 

tween  different  modes  of  operation —  ^ 

between  regular  and  chaotic  firing,  for 

example — depending  on  subtler  fac-  1 3.14  = — -  r 

tors.  These  include  not  just  the  value  of  a  |  v  ( O 

particular  train  of  incoming  signals,  but  f 
also  the  interaction  between  many  in-  ^1.57- 
coming  signals  and  the  neuron’s  recent  pfir  , 

history.  So  far,  Farhat  has  made  only  ^-kM  [ 

single  neurons,  and  he  hasn  t  linked  °00  520  540  560 '  g80  600  620  640  660  680  1 
them  together  into  a  complete  network.  Driving  Signal  Frequency  (Hertz) 

But  his  latest  simulations  suggest,  he  says,  Nervous  behavior.  The  bifurcation  neuron 
that  they  could  yield  neural  nets  with  switches  among  many  different  firing  modes,  de- 
more  lifelike  behavior  than  has  been  pending  on  the  frequency  of  the  signal  it  receives, 
seen  in  networks  to  date,  such  as  the 

ability  to  see,  recognize,  and  even  react  to  the  world  in  real  time. 

Whether  he  will  succeed  is  still  an  open  question,  says  Daniel 
Collobert,  a  neural  net  expert  at  France  Telecom.  But  he  notes 
that  Farhat’s  bifurcating  neuron  provides  a  level  of  behavioral 
complexity  “that  [artificial]  neural  networks  could  not  previously 
[show],  and  1  guess  never  will,  because  of  the  functional  simplicity 
of  [their]  neurons.” 

The  neurons  in  traditional  nets  sacrifice  important  informa¬ 
tion,  because  they  know  only  how  many  spikes  reached  them  in  a 
given  period,  but  not  when  each  spike  arrived.  To  an  ordinary 
neuron,  the  periodic  signal  110110110  (where  a  1  is  a  spike) 
would  be  exactly  the  same  as  101101101. 

Yet  real  neuronal  nets,  in  the  brain,  capture  this  timing  infor¬ 
mation.  A  neuron  fires  because  incoming  signals  cumulatively 
depolarize  the  excitable  membrane  of  the  neuron’s  output  device, 
the  axon.  Afterward,  there  follows  a  period  when  the  membrane 
cannot  respond  at  all  to  an  incoming  signal,  which  gives  way  to 
another  slow  buildup.  A  pulse  arriving  immediately  after  firing 
will  have  a  completely  different  effect  on  the  output  of  the  neuron 
from  one  that  arrives  immediately  before. 

Real  neurons  also  respond  differently  to  signals  when  they  are 
correlated  than  when  they  arrive  separately.  Farhat  explains  that 
signals  arriving  simultaneously  through  different  dendrites  produce  a 


periodic  modulation  of  the  neuron’s  electrically  excitable  membrane, 
just  as  two  beams  of  light  from  the  same  source  produce  a  periodic 
pattern  of  light  and  dark  patches  when  they  interfere  with  each  other. 
The  oscillation  modulates  the  neuron’s  response  to  later  inputs. 

Conventional  digital  circuitry,  with  its  arrays  of  on-off  switches, 
can’t  efficiently  mimic  this  kind  of  behavior.  So  Farhat  has  been 
combining  resistors,  capacitors,  and  other  components  into  so-called 
analog  circuits,  which  can  adopt  any  intermediate  state  between  “on” 
and  “off.”  One  proof-of-principle  design  incorporated  two  capacitors, 
which  charge  up  in  parallel  as  incoming  signals  build  up.  Eventually, 

_  one  capacitor  “breaks  down,”  allowing 
|  current  to  flow,  which  switches  on  a  light- 
|  emitting  diode.  The  diode  discharges  both 
uj  capacitors,  reversing  the  breakdown  and 
|  allowing  the  charging  to  begin  again.  This 
w  behavior  makes  the  circuit  time-sensitive: 

!  Signals  arriving  when  the  capacitors  have 
just  been  discharged  have  a  different  effect 
^ ;;  ,  from  signals  arriving  earlier  or  later. 
d||  ;  The  latest  incarnations  of  Farhat’s  neu- 

$  j  rons  display  more  complex  behavior  (see 

fk  ffe  diagram).  When  many  neural  inputs 

'  ,  / ,  /  .  (spike  trains)  arrive  at  the  same  time,  they 

■equency  (Hertz)  660  68°  7°°  interact  to  generate  an  electrical  oscilla- 
furcation  neuron  tion,  which  affects  the  neuron’s  firing, 

■rent  firing  modes,  de-  Small  changes  in  the  frequency  of  this 
)f  the  signal  it  receives.  oscillation,  caused  by  varying  input  sig¬ 
nals,  produce  huge  shifts  in  the  output 
behavior.  For  instance,  oscillation  frequencies  below  about 
550  hertz  produce  periodic  firing  with  two  spikes  per  cycle.  Just 
above  this  frequency,  the  output  rapidly  changes  to  chaotic  firing. 

Farhat  and  his  colleagues  are  now  planning  to  link  such  neurons 
into  a  full  array  using  optical  signals,  which  should  allow  them  to 
create  the  dense  thicket  of  interconnections  needed  for  a  large 
neural  net.  In  a  paper  to  be  published  in  the  Journal  of  Intelligent  and 
Robotic  Systems  early  next  year,  Farhat  describes  a  computer  simula¬ 
tion  that  offers  a  glimpse  of  how  such  a  network  would  behave.  The 
bifurcation  neurons  seem  to  form  “netlets” — subsets  of  the  neurons 
that  work  together.  In  an  even  more  recent  simulation,  Farhat 
found  that  the  netlets  formed  a  kind  of  neuroanatomy,  with  differ¬ 
ent  clusters  of  netlets  responding  to  stimuli  from  different  sources  in 
the  environment.  “That’s  exactly  the  same  as  people  observe  when 
they  look  at  functional  MRI  [magnetic  resonance  imaging]  and 
PET  [positron  emission  tomography]  scans  of  the  brain,”  Farhat 
says.  “Depending  on  the  inputs,  the  stimulus  from  the  outside,  or 
the  cognitive  task  that  the  person  is  engaged  in,  we  see  different 
parts  of  the  brain  firing.”  It’s  that  kind  of  complexity,  Farhat  thinks, 
that  could  make  his  networks  of  bifurcation  neurons  capable  of 
simple  abilities  that  we  take  for  granted.  -Sunny  Bains 

Sunny  Bains  is  a  science  and  engineering  writer  in  Edinburgh ,  U.K. 


mimic  some  aspects  of  the  brain  in  so-called 
neural  nets,  networks  of  “processors”  linked  by 
“synapses” — connections  that  strengthen  or 
weaken  depending  on  activity,  enabling  the 
net  to  learn  from  experience.  But  these  neu¬ 
ral  nets  generally  aren’t  real  physical  de¬ 
vices — instead,  they  are  simulated  ones,  run¬ 
ning  as  software  on  conventional  computers. 


What’s  more,  their  neurons  are,  with  few  ex¬ 
ceptions  (see  sidebar),  generally  much  sim¬ 
plified  versions  of  the  real  thing.  Neuro- 
morphics,  on  the  other  hand,  is  an  effort  to 
capture  some  of  the  richness  of  actual  neurons 
in  hardware — transistors,  capacitors,  and  re¬ 
sistors,  all  fabricated  onto  silicon  chips — in 
what  is  called  analog  VLSI,  or  simply  AVLSI. 


Besides  allowing  transistors  to  operate  at 
many  different  voltage  levels,  neuromorphic 
engineers  are  designing  them  to  serve  as  both 
calculation  and  memory  elements.  Work  by 
Lance  Glasser  at  the  Massachusetts  Institute  of 
Technology,  and  by  Mead  and  his  team  at 
Caltech,  has  led  to  the  design  of  a  new  type  of 
transistor,  the  floating  gate  transistor,  which 
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